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Channel Bonding 


| am trying to channel bond 4 gigabit NICs on the compute nodes to 

(theoretically) improve performance, but am having some difficulty doing it correctly. 

| can easily add the configuration by hand, but then if a nodere-installs, it would be lost. 

| have experimented withthe rocks add hostinterface command, but cannot seem to 
figure out how to tell it to enslave the other ethernets. What is the correct way to do this? 


Hello all: 
Has anyone done channel bonding before on Rocks? Detailed instruction would be helpful. 


It was mentioned the other day that ROCKS does not support the bonding network cards (NICs). 
| was a little confused by this as CentOS certainly does as a subset of RedHat Linux (or so it appears). 
Would someone be kinda enough to fill me in on what aspect of ROCKS does not support bonding (BONDO, etc)? 


So I decided to start the next step which is to channel bond each nodes 2 ethernet cards. | 
got up to compute node 3. And decided to check things out with the previous nodes. For some 


reason, rsh is now refusing connections from the headnode to the channel bonded compute 
nodes. It was working on ethO perfectly. What could cause this to happen? 


€ | mean, how hard can it be? 


5 With the Rocks Command Line, | figured we could do it with 3 
existing commands 
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Channel Bonding 


+ Created a new command to configure 
channel bonding 


# rocks list host interface compute-0-1 


SUBNET  IFACE MAC IP NETMASK MODULE NAME VLAN OPTIONS 
private 6520 00:le:4f:b0:74:ef 10.1.255.253 255.255.0.0 toi  compute-0-1 ---- ------- 
------- ethl 00:10:18:31:74:43 ------------ ----------- tg3 ----------- ---- ------- 


+ Run the command: 


# rocks add host bonded compute-0-1 channel=bond0 interfaces-eth0,ethl ١ 
ip=10.1.255.253 network=private 
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Channel Bonding 


€ The result 


# rocks list host interface compute-0-1 


SUBNET  IFACE MAC IP NETMASK MODULE NAME VLAN OPTIONS CHANNEL 
private bond) ----------------- 10.1.255.253 255.255.0.0 bonding compute-0-1 ---- ------- ------- 
se etho 007 Tei 4£:b0-74:ef SEE STD ры 3 жшк ZE4WER-EBond9 
קךא-צשר‎ ethi @ OOF 105 19 231-74-43 ЕЕ ЕТЕ ИЕ ЕТЕ Е COS ЕЕЕ ЕЕЕ ee Бола) 


+ Can apply the change חס‎ the fly: 


# rocks sync config 
# rocks sync host network compute-0-1 
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© Firewall Configuration via the 
Rocks Command Line 


+ But, for channel bonding to work, we 
needed to make the firewall adapt to the 
configured interfaces 


INPUT -i eth0 -j ACCEPT 
INPUT -p tcp --dport 0:1024 -j REJECT 
INPUT -p udp --dport 0:1024 -j REJECT 


-i Бопа0 -j ACCEPT 
-p tcp --dport 0:1024 -j REJECT 
-p udp --dport 0:1024 -j REJECT 





© 2010 UC Regents 


© Firewall Configuration via the 
Rocks Command Line 


+ Added a boat load of commands 


add appliance firewall {appliance} [action=string] [chain=string] 

add firewall [action-string] [chain=string] [network=string] 

add host firewall (host) [action=string] [chain=string] [network=string] 
add os firewall (os) [action=string] [chain=string] [network=string] 

close appliance firewall {appliance} [network=string] [protocol=string] 
close firewall [network-string] [protocol=string] [service=string] 

close host firewall (host) [network=string] [protocol=string] [service=string] 
close os firewall (os) [network=string] [protocol=string] [service=string] 
dump appliance firewall 

dump firewall 

dump host firewall 

dump os firewall 

list appliance firewall [appliance]... 

list firewall (None) 

list host firewall [host]... 

list os firewall [os]... 

open appliance firewall {appliance} [network=string] [param=string] 

open firewall [network-string] [protocol=string] [service=string] 

open host firewall {host} [network=string] [protocol=string] [service=string] 
open os firewall (os) [network=string] [protocol=string] [service=string] 
remove appliance firewall {appliance} [action-string] [chain=string] 

remove firewall [action-string] [chain=string] [network-string] 

remove host firewall {host} [action=string] [chain=string] [network=string] 
remove os firewall (os) [action=string] [chain=string] [network=string] 


report host firewall (host) 
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© Firewall Configuration via the 
Rocks Command Line 


+ The commonly used commands will be: 


rocks open host firewall (host) [network=string] [protocol=string] [service=string] 
rocks close host firewall (host) [network-string] [protocol=string] [service=string] 
rocks add host firewall (host) [action=string] [chain=string] [network=string] 


rocks list host firewall [host]... 


€ For example, to open up web access on a 
public interface: 


+ rocks open host firewall compute-0-0 network-public ١ 
protocol-tcp service=www 
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ez The Private Network Doesn't 
Have to be "ethO" 
+ But to get the new firewall configuration 


working, we had to break the "6170 = 
private network" relationship 


# rocks list host interface compute-0-0 "REI 0-1 compute-0-2 


HOST SUBNET  IFACE MAC NETMASK MODULE NAME 

compute-0-0: KC ivate bond0 --------- SÉ id 1.255.254 255.255.0.0 bonding compute-0-0 
computes 0-0-3 arn RetRON 00): бе: 07 250 ай БУЕ ЕТЕНЕ БУТ הזרא חורא‎ RET 
compute-0-0: ------- eth1 00:19:b9:21:b8:b6 ------------ ----------- ------- ----------- 
compute-0-2: zc Месно TO Of MOENS הבו‎ A We === 
compute-0-2: ER: 1866 6621 00:1e:4£:b0:72:2f 10 . 1 . 255 . 252 255.255.0.0 ------- compute-0-2 
compute-0-2: ----- eth2 00:0e:0c:5d:7e:59 
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С The Private Network Doesnt 
Have to be "ethO" 


+ Now when the “private” network cable 
moves from one interface to another, the 
“private” network configuration will follow 
the cable 

5 The interfaces will never be renamed 
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Login Appliance 
€ Administrators have asked for a node in 
the cluster that is not the frontend where 


users can login, develop and launch their 
code 


€ | mean, how hard can it be? 


© 2010 UC Regents 12 


e 





13 





Login Appliance 


Insert Ethernet Addres 
0 pened kickstart access 


Select An Appliance Type: 


Compute 
Ethernet Switch 
IPMI 


NAS Appliance 

Power Distribution Unit 
Tile 

YM Container 
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Login Appliance 


+ Created two new attributes: 


2 submit host 


* One can submit jobs to the queuing system from 
this host 


2exec host 
* Jobs can be executed on this host 
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Login Appliance 


+ Login appliance: 
2 submit_host = true, exec_host = false 
+ Compute node: 
2 submit_host = false, exec_host = true 
+ Can set/unset the attributes for any host 
2 Can easily make all tile hosts execution hosts 
2 Can easily exclude specific hosts as queuing 
System resources 
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Avalanche Installer 
Retooled 
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Avalanche Installer Retooled 


+ We went to Nebraska 
د‎ We came back humbled 
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Avalanche Installer Retooled 


+ Found several issues that limited 
scalability 


2 Some of the easier fixes made it into Rocks 
5.9 


+ Realized that we needed to get as much 
traffic off the frontend as possible 
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Ki Avalanche Installer Retooled 


Installing Node Frontend 





» vmlinuz, initrd.img 
2 -18 MB 

€ ks.xml 
> ~0.3 MB 

+ “stage 2 files” 
2 -200 MB 


ren 


| Installing Node | | Installing Node | 
| Installing Node | Node 


₪ بي‎ | Installing Node | | Installing Node | 
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Avalanche Installer Retooled 


+ The Opportunity: distribute the “stage 2 
files” with BitTorrent 


+ Had סז‎ rewrite the client-side of 
Avalanche in C 
2 Which | loved! 
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Avalanche Installer Retooled 


+ Added a “package predictor” 


5 When a client asks for a package, the tracker returns 
a list of the next 10 packages that the client will likely 
ask for 


5 Reduces the tracker load on the frontend by 10x 
+ Only assign 3 clients for each package 


2 Previous version sent back all the available clients 
for a package 


5 Reduces the tracker response message size for 
large concurrent reinstallations 


© 2010 UC Regents 21 





22 


Ki Avalanche Installer Retooled 


Installing Node Frontend 
— PXE --------------р»- 

vmlinuz, 

initrd.img 


— kickstart.cgi ————- 
я4---------К<.хті 





— EE Installing Node 


get stage2img ТТ Installing Node 


иии Installing Моде 
RPM request RPM. | Installing Node 
RPM request m" , Installing Node 
RPM request ₪ i Installing Node 
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Avalanche Installer Retooled 


+ Can have multiple “trackers” and “package servers” 


2 Previous version: only the frontend tracked and served 
packages 


€ Tracker assigns clients based on "least-recently used" 


+ Group clients by “co-op” 


5 A client will try to get a package from members of its “co-op” 
first 


2 Can set the "co-op" with an attribute 
* Default co-op is the rack id 
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Graph Traversal Fixed 
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Graph Traversal Fixed 


<edge from=“b” cond-"x11"» Profile Graph 
<to>c</to> 
</edge> 
«edge from=“c”> 
<to>e</to> сь ЖЕ 
</ейде> 


+ Original implementation had а 
major bug 
> If x11 was false, "c" would be 
omitted, but "6" would be included! 
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Profile Graph 


» Fix: prune the tree 


2 stop traversing at "b" if Съ) (a) 


"X11" is false 


Hocks Graph Fixes 
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Multiple DNS Zones 
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Multiple DNS Zone support 


+ Multiple subnets 
+ Each subnet maps to a DNS zone 
+ Serve DNS for multiple interfaces 


+ Customize zone names 
٠ Private network need not be “.local” 
+ Examples 
٠ Optiputer network - <hostname>.optiputer.net 


e Local network - «hostname» . local 
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Multiple DNS Zones 


# rocks add network optiputer 192.168.0.0 255.255.0.0 \ 
servedns-true dnszone=myri 


# rocks list network 


NETWORK SUBNET NETMASK MTU DNSZONE SERVEDNS 
private: 10.1.0.0 255.255.0.0 1500 local True 
public: 137.110.119.0 255.255.255.0 1500 rocksclusters.org False 
optiputer: 192.168.0.0 255.255.0.0 1500 myri True 


+ rocks sync dns 
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Updates 


30 
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Software Update 


+ Support for Rocks published updates 
2 Patches 
2 Security fixed 
2 Rocks and/or CentOS packages 


+ Not the same problem as general 
software update 
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# rocks update 


Downloads new packages from 
صم م و‎ uc كات‎ for your release of Rocks 
Runs any update shell scripts we provide 


Removes any update XML files for Rolls you 
don't have 


Creates an Update Roll 
Create an Update Yum Repository 


Does a "yum update" on fronted using only this 
repository 
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What About Compute Nodes? 


+ You now have an Update Roll 
5 Enable the Roll 
د‎ Rebuild the distribution 
+ Pick One 
> Re-install nodes 
> Run “yum update” on nodes 
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"rocks run host” Retooled 
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"rocks run host” Retooled 


Now 100% tentakel free! 


rocks run host [host]... (command) [collate=string] 
[command=string] [delay=string] 
[managed-boolean] [stats-string] [timeout=string] 
[x11-2boolean] 


Collate: prepend the name of the host on each line of the output 
Delay: delay X seconds between command launches 


Managed: only execute on “managed” nodes (e.g., not NAS 
appliances) 
> A managed host has the attribute “managed” set to "true" 


otats: print how long it took to run each command 
Timeout: terminate after X seconds 
X11: set to 'סח'‎ to disable X11 forwarding 
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Features We're 
Considering 
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Features We're Considering 


+ Console access to virtual compute nodes 
from within a virtual frontend 


2 Helated: 


* Allow users to power on/off VMs from within 
virtual frontend 


+ Multiple distribution support 


+ Global/OS/Appliance/Host hierarchy 
cleanup 
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Features We're Considering 


+ Roll "personalities" 


2 A method to select several rolls by clicking 
one checkbox 


+ Making the SGE job queue data collection 
more efficient 
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Features We Should 
Consider? 
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